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1.0  Abstract 

Research  into  one  of  the  possible  ' pproaches  to  thq 
Investigation  of  optimum  analytical  and  .'orecasting 
techniques  in  weather  analysis  and  forecasting  was 
initiated  on  1 July  1952.  In  effect,  this  approach 
consists  of  withholding  from  the  meteorologist  certain 
combinations  of  synoptic  data,  yet  requiring  him  to 
forecast  all  the  usual  weather  parameters  Including 
those  withheld. 

During  the  time  that  the  necessary  charts  were 
being  prepared  for  the  main  project,  a pilot  test  was 
conducted  for  the  purpose  of  determining  the  character- 
istics of  the  experimental  design.  The  results  of  the 
pilot  test  for  sea-level  and  500-mb  data  indicate 

(1)  that  an  effective  deterioration  of  forecast  scores 
is  noticeable  only  for  severely  reduced  information; 

(2)  that  variation  of  the  forecast  verification  schemes 
merely  shifts  the  error  level  of  forecasts  without 
affecting  the  score  trends;  and  (3)  that  there  exists 
an  optimum  scheme  of  reduction  of  data  furnished  to  the 
meteorologist  which  results  in  a considerable  saving 

of  time  required  for  spotting,  analyzing,  and  forecast- 
ing, without  materially  affecting  forecast  scores. 


In  addition,  an  experiment  was  conducted  with  a 
class  of  meteorology  trainees;  one  section  of  this 
class  was  required  to  forecast  from  previously  prepared 
prognostic  charts  that  actually  represented  correct 
analyses  of  the  subsequent  maps;  another  section  of  the 
class  was  required  to  construct  their  own  prognostic 
charts.  The  results  of  this  experiment  cannot  be 
considered  conclusive  at  this  time. 


2,0  Personnel  and  Administration 

2,1  The  following  persons  were  employed  on  the 
project  daring  the  period  in  question:  Dr.  H.  Nouberger, 

?ro^«ssor  of  "etocr'^l'^gy  and  Project  Supervisor, 
approxiraatoly  l/3-tlme  without  coat  to  the  project; 
wi*.  V.  Moyer,  Research  Associate  and  Assistant  Project 
Supervisor,  full-time;  Mr.  I.  Van  der  Hoven,  Research 
Assistant,  full-time  after  1 September  1952;  and 
Mr.  David  L.  Jones,  Graduate  Assistant, 

Mr,  Howard  Simmons,  Supervisory  Meteorologist,  U.  3. 
Weather  Bureau,  Evansville,  Indiana,  is  participating 
in  the  analysis  and  forecasting  of  the  final  test  data 
without  cost  to  the  project.  Dr.  Hans  Panofsky, 
Associate  Professor;  Dr.  Charles  L.  Hosier,  Jr,, 
Assistant  °rofessor,  and  Mr,  Donald  G,  Yerg,  Instructor 


2 


in  Meteorology,  assisted  from  time  to  time  in  the 
capacity  of  consultants  without  cost  to  the  project. 


2.2  The  following  special  students  and  upper 
classmen  served  as  technical  assistants  on  a part- 
time,  hourly  basis: 


Ifr.  James  ?.  Anderson 
Mr.  Floyd  C.  Elder 
Mr.  William  Holtzman 
Mr,  Lowell  Krawitz 
Mr.  Harry  R.  Mansfield 
Mr.  Evan  J.  Tibbott 
Prodlpto  Roy 
Mr,  Robert  B.  Wassail 


2.3  The  secretarial  work  was  performed,  until 
1 September  1952,  by  Mrs,  Kay  Fisher  at  no  expense 
to  the  project.  After  1 September,  Mrs,  Mary  Wagner 
was  engaged  as  full-time  project  secretary;  she  was 
assisted  from  time  to  time  by  Mrs,  Peggy  Rase,  without 
cost  to  the  project. 


3.0  Conferencos 

3.1  Between  23  and  27  September  1952,  Dr.  Nouborgor 
and  Mr.  Moyer  undertook  a trip  to  Cambridge,  Massa 
chusetts,  and  Now  York  City  for  the  purpose  of  conferr- 
ing with  ProfoBSor  0.  P.  Wadsworth,  Dr.  J.  0.  Bryan, 
and  Mr.  William  Paulson,  at  Massachusetts  Institute 
of  Technology,  and  with  Professor  James  E,  Miller,  at 
New  York  University,  on  the  problem  of  the  accuracy 
of  short-range  weather  forecasts,  the  limitations  of 
weather  vorlflcation  systems,  ana  the  philosophy  of 
non-lsobarlc  analysis.  While  in  Cambridge,  the  above 
project  members  also  visited  the  Mr  Force  Cambridge 
Research  Center  of  tho  Geophysical  Research  Directorate 
where  they  mot  v/ith  Drs,  Richord  A,  Craig  and  William 
K.  Wldger,  and  Messrs,  Irving  I.  Grlngorten  and  Ivor 
Lund  in  a discussion  of  forecast  verification-  The 
results  of  the  conferences  con  be  summarized  as 
follows:  (1)  Tho  handicap  of  reduced  Information, 

particularly  the  omieslon  of  barometric  pressiire  fr  a 
surface  weather  maps,  seems  to  be  more  psychological 
than  physical,  Tho  amount  of  information  currently 
presented  is  based  on  tradition  rather  than  on 


scientifically  well  considered  needs.  As  e resiilt, 
the  experienced  forecaster  revolts  at  the  forced 
task  of  forecasting  from  anything  but  the  complete 
map  to  vdilch  he  is  accustomed;  (2)  two  general 
classes  of  forecast  verification  systems  can  be 
consldored:  the  multiple  error  point  system  in  which 

one  or  more  error  points  are  scored  depending  on  the 
degree  by  which  the  forecast  item  was  missed;  or  the 
binary  system,  in  which  an  item  forecast  is  either 
right  or  wrong.  The  latter  system  appears  to  be 
preforaola,  notably  from  the  point  of  view  of  utility 
and  of  minimizing  bias.  This  supports  our  original 
position  in  this  matter. 

3.2  Mr.  David  Jones  visited  the  National  Weather 
Records  Center,  Asheville,  North  Carolina,  on  29 
August  1952,  while  on  annual  leave.  Through  the  active 
assistance  of  ‘.?r.  Lesley  Smith,  Supervisor,  he  was 

able  to  obtain  the  scattered  data  that  were  missing  from 
the  Ozalid  V/BAN  Analysis  Center  maps  which  were  used  as 
source  of  synoptic  information. 

3.3  LCDR  Donald  R-  Jones,  AROM  Project  Officer, 
visited  the  project  in  State  College  on  $ September 
1952  for  the  purpose  of  discussing  procedures.  His 
visit  was  followed  by  telephone  conversations  both 
with  him  and  with  LCDR  William  J.  Kotsch  about  mis- 
understandings over  the  over-all  purpose  and  procedures 
of  the  experiment,  as  well  as  contract  details. 
Considorable  correspondence  was  required,  subsequently, 
in  connection  with  ordering  supplies  and  materials. 


Ip.O  Investigations  being  undertaken 

Prior  to  the  1 July  1952  start  of  the  work,  complete 
plans  had  been  formulated  to  put  the  project  into 
operation  on  the  starting  date  without  delay.  An 
office.  Room  3l8,  was  set  aside  in  the  Mineral  Industries 
Building  for  tho  sole  use  of  project  personnel.  In 
addition  to  tho  usual  equipment  and  tho  necessary  minor 
supplies,  it  was  furnished  with  six  drafting  tables  anJ 
a portQDie  light  table  to  exiiodito  plotting  and 
analysis  of  maps.  This  offioo  serves  as  permanent 
quarters  for  Messrs.  Moyer,  Von  der  Hoven,  and  Dav  . 
Jones,  and  provides  working  space  for  part-time  tech- 
nical assistants. 

The  preliminary  planning  included  tentative 
decisions  on  the  stations  for  which  forecasts  wore  to 
bo  made,  forecast  items,  verification  systems  and 
tolerances,  and  the  specific  raotoorological  situations 
to  be  tested.  Dr.  Hosier  assisted  Dr.  Noubergor  in 
choosing  representative  winter  and  summer  synoptic 
situations  for  testing.  Tho  other  project  personnel 


were  not  consulted  during  this  choice,  since  It  was 
desired  to  keep  them  In  Ignorance  of  even  the  broad 
features  of  the  synoptic  maps  that  they  would  later 
bo  required  to  analyze  and  prognosticate. 

Prom  the  multitude  of  possible  "schemes”  of 
synoptic  weather  data  presentation  (factorial  12), 
five  logical  combinations  were  chosen  for  testing  In 
a Pilot  tost  prior  to  the  start  of  the  final  analysis 
of  the  two  selected  situations.  Dr.  Panofsky  and 
Mr.  Yerg  were  consulted  In  the  choice  of  these  schemes 
because  of  the  statistical  implications  Involved  in  the 
future  analysis  of  the  results. 

Also,  I4r.  Moyer  developed  a new  base  map,  of 
convenient  size  and  scale,  that  would  Include  the  fore- 
cost area  as  well  os  an  adequate  "influence  zone"  to 
the  west  of  this  area  (Enclosure  l).  This  map  was 
printed  by  offset  process  in  sufficient  quantity  for 
the  duration  of  the  project. 

Because  of  the  magnitude  of  the  program  with 
consideration  of  time  consumption  In  spotting,  analyzing, 
and  forecasting  from,  the  test  maps,  the  forecast  stations 
were  restricted  to  the  area  between  25°  N and  65®  N 
latitude  and  30°  W and  Q8®  W longitudo.  However,  the 
Influence  zone  was  extended  to  110°  W longitude. 

Within  this  area,  the  forecast  stations  selected  were: 

(1)  Weather  ship  "Alpha,"  IfYA 

(2)  Weather  ship  "Beta,"  IfYB 

(3)  Moosonoo , Ontario,  836 

(I4.)  Woathor  ship  "Coca,"  l^-YC 

(5)  International  Palls,  Minnesota^  ?47 

(6)  Caribou,  Maine,  712 

(7)  Torbay,  Newfoundland,  8OI 

or 

Popporroll  APB,  Newfoundland,  I98 

(6)  Chicago  (Joliet),  Illinois,  S3k-  (JOT) 

(9)  Buffalo,  New  York,  528 

(10)  Nantucket,  Massachusetts,  5o6 


s. 


(11)  Weather  ship  ’’Delta,”  lj.YD 

(12)  Kansas  Citv.  Mlijcourl  (Fort  Leavenworth, 
Kansas),  1440  (FLV) 

(13)  Nashville,  Tennessee,  327 
(14.)  Ha tt eras.  North  Carolina,  304 

(15)  Shreveport  (Barksdale  AFB),  Louisiana, 

2I4.8  (BAD) 

(16)  Pensacola,  Florida,  222 

(17)  Klndley  APB,  Bermuda,  016 

(18)  leather  ship  ’’Echo,”  4^ 

(19)  Brownsville,  Texas  , 250 

(20)  Miami,  Florida,  202 

The  stations  listed  in  parentheses  are  the  upper  air 
sounding  stations  nearest  the  indicated  surface  stations. 

Although  it  had  originally  been  anticipated  that 
I.B.M.  methods  would  be  used  to  analyze  the  results 
of  the  tests,  these  procedures  wore  found,  after 
consultation  with  College  experts,  to  bo  too  inflexible 
for  our  purpose;  therefore,  McBee  Keysort  cards  wore 
designed  to  facilitate  subsequent  analysis  (Enclosure  II) 

4»1  Pilot  test.  During  th»:  early  weeks  of  July 
all  project  members  »;ore  occupied  in  plotting  the  data 
for  a pilot  test  which  was  designed  to  tost  the 
experimental  schemes  of  data  presentation  and  analysis. 
Teletype  data  were  used  for  this  test,  there  beln  lx 
maps  analyzed  for  each  scheme.  The  situation  analyzed 
covered  tho  period  from  1235Z  of  1 July  1952  through 
0035z  of  4 July  1952,  the  continuity  followed,  thus, 
consisting  of  12  hours. 

For  purposes  of  plotting  ease,  tho  synoptic  code 
was  rearranged  as  follows: 

(1)  Land  Stations:  iiidd  ffNWw  Ct Nj.h-CjyiCjT 

4VVTjjT^  TTPPP  appWR^  7RRDc® 

(2)  Sea  Stations:  LLL  dd  ffiiww  ^L^h^c^M^H 

4VVT^Tjj  TTPP?  appWR^  DgV^ 


final 
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Also,  mimeographed  forms  were  used  which  contained  the 
reporting  stations  in  a given  order  to  facilitate 
spotting  with  Ditto  inic:  i.e.,  the  stations  v/ore 

listed  from  within  latitude  bands  running  from  west 
to  oast,  starting  at  the  north  of  the  map,  such  that 
the  spotter  could  avoid  smudging  data  already  spotted 
and  not  yet  dry. 

The  following  iichenes  were  chosen  for  providing 
synoptic  data: 


E. 

lii// 

///vvy/ 

/ / / / / 

k///  / 

TTPPP 

/////  ///// 

D. 

iii// 

//Nww 

k///  / 

TTPPP 

app//  ////  / 

C. 

iiidd 

: ffNww 

4-WT^T^ 

TT/// 

appWR^YRR// 

B, 

iii// 

//Nww 

<=l/  / =;.:Ck 

4VVT^T^ 

TTPPP 

appWR^72iy^ 

A. 

iiidd 

ffNww 

a a 

TTPPP 

aop”T^.  7RRD  s 
t c 

All  six  naps  of  each  scheme  wore  analyzed  and 
prognosticated  before  work  .vit'-  the  next  scheme  was 
started.  The  schemes  were  analyzed  in  reverse  order 
viz,,  scheme  "E”,  scheme  ”D”,  etc,  ’Ahile  the  technical 
assistants  were  occupied  in  transcribing  data  for 
the  main  test,  from  the  Ozalld  copies  of  V-’BA}!-!  sea- 
level  and  500-mb  analyses  of  the  ‘'BAK  Aiialysi;?  Center 
supplied  to  us  by  LCD3  Donald  R,  Jones,  Messrs, 

Moyer  and  David  Jones  conducted  the  pilot  experiment. 
Neither  of  those  ">en  cor. oared  analyses  during  the  test, 
nor  were  tholr  forocaste verified  until  all  had  been 
submitted.  Each  man  drew  from  his  personal  experience 
in  analysis  and  foi’ocasting  and  used  whatever  device 
was  possible  within  the  limitations  of  each  given 
scheme.  No  attempt  was  nede  to  force  either  man  to 
use  any  of  the  so-called  standard  methods  of  prognosis. 
That  is  to  say,  if  one  of  tho  forecasters  chose,  for  a 
particular  map,  to  be  complotoly  subjective  and  to 
rely  entirely  upon  intuition,  he  v.-as  at  liberty  to  do 
so;  on  the  other  hand,  if  h'^  f-itr.Anntnd  to  apply 
objective  forecasting  rules  and  methods  in  his  pr  _..oSis 
again  he  was  free  to  do  so. 

Because  of  our  desire  to  accomplish  thia  test 
within  a mini  um  of  time,  only  incomplete  upper  air 
charts  were  at  first  -ivailablo;  they  consisted  of 
500-mb  charts  cf  irregular  continuity,  drawn  by 
students  in  their  s;«-noptic  laboratory  course.  However, 
after  it  becai-’o  appeorent  that  the  pilot  teat  could  be 


completed  well  within  our  self-imposed  desdline,  ei 
similar  experiment  was  conducted  with  500»mb  data  for 
the  same  period,  with  charts  specially  prepared  for 
this  purpose.  The  five  schemes  of  data  presentation 
at  this  level  were: 


E.  ilidd 

ff/// 

//// 

D.  ill// 

//hhh 

///  / 

C.  llldd 

ff/// 

TT/  / 

B,  Hi// 

//hhh 

tt/  / 

A.  llldd 

ffhhh 

TTT  T 

d d 


The  procedure  in  this  phase  of  the  pilot  experiment 
WQs  precisely  the  same  as  that  for  the  sea-level  data. 

All  six  charts  of  scheme  ”E"  were  analysed  and 
prognosticated  before  the  forecaster  turned  his  attention 
to  scheme  "D”,  etc.  During  this  phase,  the  previously- 
analyzed  ”A’‘  scheme  of  sea-level  maps  was  used  as  supole<- 
mentary  information. 

No  tlrao  limits  or  deadlinee  were  imposed  upon  the 
forecasters  during  the  test  run.  Since  a time  study  was 
conducted  for  this  experiment  on  the  same  basis  as  that 
planned  for  the  final  test,  it  was  believed  essential 
to  lot  each  individual  set  his  own  pace.  Also,  because 
the  personality  of  the  foi'ocastor  certainly  enters 
into  the  facility  with  which  ho  arrives  at  a forecast, 
the  results  obtained  wore  not  biased  because  of  this 
factor.  Every  effort  was  made  to  achieve  realism  dur- 
ing the  process  of  the  experiment,  except  that  it  w-^s 
not  possible  for  the  forecaster  to  determine  the  trend 
of  verification  of  his  forecasts,  as  would  be  the  case 
during  normal  weather  station  operation.  This  exception 
is  unfortunate,  perhaps,  since  it  did  not  permit  normal 
adjustments  of  analyses  or  forecasts  in  conformity  with 
developments  in  the  synoptic  pattern:  on  the  other  hand, 
this  practise  resulted  in  an  investigation  of  minimum 
operating  procedures,  since  the  forecasters  were  required 
to  work  under  what  wu  can  assume  to  bo  the  extreme 
possible  handicaps. 

To  accelerate  accomplishment  of  the  goal  of  the 
pilot  tost,  only  ten  of  the  forecast  stations  named 
above  were  considered.  These  ware: 

(1)  Interna tlbnal  Palls 

(2)  Kansas  City  (Port  Leavenworth) 
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(3)  Nashville 

(1^)  Batteras 

(5)  Iflaml 

( 6 ) Bermuda 

(7)  Ship  "Echo" 

(8)  Caribou 

(9)  Ship  "Delta" 

(10)  Ship  "Alpha" 

Sea«>levol  forecast  items  included  (1)  wind  direction 
and  speed,  (2)  sky  condition  or  present  weather, 

(3)  ceiling,  (I4.)  visibility,  (5)  special  phenomena 
(fog  or  thiinderstorm) , (6)  precipitation  amount, 

(7)  temperature,  and  (3)  dow  point,  500-mb  forecast 
Items  included  (1)  height-change,  (2)  temperature- 
change,  and  (3)  wind  direction  and  speed.  Porecarts 
wore  verified  on  the  basis  of  slight  modification 
the  system  suggested  in  Pulk  and  Murphy,  Workbook  for 
Weather  Forecasting.  Prentice  Hall,  New  York,  19i>0, 

k,!!  Results  of  the  Pilot  Tost 

The  main  results  of  the  pilot  test  for  the  surface 
data  are  as  follows; 

(1)  The  maximum  difference  in  toore  between 
the  two  forecQstors  for  a given  time  and  scheme 
was  22%f  whereas  the  maximum  difference  between 
station  forecasts  for  one  forecaster  was  375^» 
Table  I gives  the  ranges  of  the  per  cent  errors 
of  the  total  scores  for  each  forecaster  and  for 
various  schomos, 

Taoio  i.  OX  pCX*  ^ Ox  ^ O^cajl 

surface  scoros  f or  t wo  forecasters 
and  various  schemes. 


Schemes 

Forecaster 

..I  ^ 

Porocastor 

II 

Both 

Porocastors 

A 

16-k^ 

4341 

iTjqn 

B 

20-k3 

4-32 

12-34 

C 

23-^ 

^1 

15-ja 

D 

16-iO 

9-38 

12-42 

E 

18-48 

io-4o 

14-42 

^ ' — It- 


' --  ' -w,  ■'■ 

J ■ vO>.  f ^ , •" 

» ^ /•' ** 

- \ K •■*  . “•^'7 

■ \ >.K:‘' 

-s  A *»' 

-’  ><'-/  ‘ ■'"’,>  yV  ''7r  */ 

V,.  ^ 

. • > .-6^' 


">0'% 


V . 

/\ 


.vys^vA,/  r^«< 
■ ^ 


/ X 


\ y'  \ 


'V 


¥4^ld£' 

X ',  L.  I iH 


Forecast  stations  for  pilot  test 
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(2)  The  difference  In  score  between  12-and 
24-hour  forecasts  for  each  forecaster  and  for 
both  forecasters  was  negligible  as  compared  to 
the  difference  between  other  parameters  such 
as  stations  and  forecast  Item.  The  maximum 
difference  In  score  for  forecaster  I was 
and  for  forecaster  II,  175^.  Table  II  gives 
the  average  per  cent  errors  for  the  various 
schemes  for  both  12-and  2i{.“hour  forecasts. 

The  numbor  of  forecast  Items  per  man  per 
schomo  was  432  for  12-hour  and  44^  fo**’  24 -hour 
forecasts.  The  total  number  of  a urf ace  Items 
forecast  by  each  man  was  4360* 

Table  II.  Average  per  cent  errors  for 

various  schemes  and  for  13>and 
24-hour  surface  forecasts* 


1 Forecaster  I 

Forecaster  II 

Both  Forecasters  | 

■HHii 

12  hr 

fmmsm 

A 

26 

WM 

24 

26 

B 

31 

20 

20 

26 

24 

C 

32 

EH 

22 

28 

27 

29 

D 

32 

HEH 

28 

27 

30 

29 

E 

33 

Ji 

— 

32 

u 

(3)Tlio  effect  of  the  scheme  on  the  score  for 
various  stations  was  very  Irregular, 
although  the  scores  for  o of  10  stations 
wore  best  for  scheme  B.  The  forecasts  for 
3 of  the  4 remaining  stations  were  stc  ^nd 
best  for  this  same  scheme.  Table  III  shows 
the  Average  cent  errors  for  various 
stations  and  schemes,  with  the  best  score 
for  each  station  underlined. 


Table  III.  Average  per  cont  surface  forecast 
errors  for  various  stations  and 
3 shorn''  ^ ■ 


Scheme 

STATION  n 

UMBKI 

Average 

747 

712 

446 

327 

304 

202 

— 

0l6 

4ya 

!• 

1 

IfXiL 

1 cr  oxx 
stations 

■"  'A  " 
B 
C 
D 
E 

1 

20 

22 

26 

‘2T-I 

25 

\'28" 

27 

26 

I? 

JO 

29 

-22 

H 

w^m 

H 

icr 

I? 

15 

12 

JL- 

2^- 

25 

27 

30 
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Figure  3 shows  graphically  the  pronounced 
trend  toward  acore  deterioration  that  exists 
with  reduced  information  for  stations  0l6, 

414.6,  304.,  and  712.  There  Is  very  little 
trend  for  stations  327,  202,  4-^D,  and  4-^* 
while  stations  74-7  and  4>YA  show  Irregular 
score  variations*  However,  It  can  be  noted 
that  those  stations  with  low  error  scores 
show  a more  significant  trend  toward  score 
deterioration  w 1th  reduced  Information,  than 
those  with  high  error  scores. 

For  a fvirther  appraisal  of  the  effect  of  the 
various  schemes  on  the  forecast  scores,  the 
frequencies  of  four  groups  of  per  cent  errors 
were  detorminod  for  the  several  schemes. 

Figure  4 shows  this  relative  frequency  dlstrl- 
butlon^  ^ significant  deterioration  In  acore 
Is  evident  only  for  schemes  D and  E where  the 
total  frequency  of  the  two  highest  error 
groups  Is  4-5^  and  655^,  respectively. 

(4)  The  effect  of  the  schemes  on  scores  for 
various  forecast  l*’cms  is  strong  only  for 
present  weather  (WW),  tomperqture  (TT),  and 
dew  point  (T^j)  , v/ith  a sli^t  effect  for 
wind  force  and  no  effect  for  visibility 

(W),  wind  direction  (DD),  and  rainfall  (RR), 

For  cloud  heights  ( hehc)»  ^be  forecast  seems 
to  improve  slightly  with  reduced  information; 
but  this  result  cannot  bo  considered  significant* 
Table  IV  summarizes  the  effect  of  the  schemes 
on  the  forecast  items.  The  underlined  vc.^  cs 
represent  schemes  which  do  not  include  the 
parameter  listed  above  the  indicated  value. 

Table  IV.  Effect  of  schemes  on  per  cent 

error  score  for  various  forecast 
items. 


jschsme  II  DD  j 

nn 
c r 

Win 

"o“o 

Vv 

HR 

- . 

TT 

^d^d 

" — 

Average 

A 

19 

27 

38 

35 

12 

29 

17 

27 

26 

B 

20 

2k 

37 

37 

13 

26 

17 

25 

25 

n 

u 

35 

.11 

30 

2k 

28 

D 

29 

2^ 

7 

32 

29 

p 

30 

E 

s 

2i 

4.8 

25 

3b 

32 

PHEQUBNCY  (%) 


0 - 15  16-30  31  - kS  14.6  - 100 

FORECAST  ERROR  CROUPS 
(Errors  in  %) 


Fig.  4.  Frequency  DistrlDutiona  of  Surface  Forecast 
Errors  (Errors  In  %) , Schemes  A Through  E. 
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(5)  The  effect  of  the  verlflcctlon  systor.  on 
the  results  as  presented  In  Table  IV  was 
tested  by  employing  a second,  slightly 
more  rigorous  verification  system"  Figure  5 
clearly  shows  that  a more  rigorous  verifica- 
tion system  increases  the  error  score 
without  materially  affecting  tho  trend 
introduced  by  the  schemes* 

Prom  tho  above  results  tho  following  conclusions  are 

drawn  and  discussed: 

(1)  Tho  scores  of  tho  two  forecasters  and 
of  12-  and  2l4.-ho\ir  forecasts  can  be  taken 
together  and  treated  as  one  statistical 
population*  This  appears  to  be  in  agroGmont 
with  the  results  presented  by  G,  P,  Wadsworth* 
Whether  or  not  this  will  still  hold  in  tho 
main  tost  of  the  project  when  more  forecasters 
are  employed  whoso  backgrounds  are  no  longer 
as  homogenoous  as  in  the  pilot  test  will  have 
to  be  inrostigatod  by  moans  of  the  main 
forecast  material, 

(2)  In  general,  tho  trend  toward  score 
deterioration  with  reduced  information  is  more 

evident  for  stations  with  low  error  scores 
than  for  those  with  high  error  scores.  This 
may  appear  to  bo  a trite  result,  because, 
e*g*,  in  the  absurd  case  of  completely  wr.ng 
forecasts  for  scheme  A,  the  score  could  not 
possibly  detorlorate  with  reduced  information. 
However,  it  can  bo  seen  from  Table  III  that 
evon  tho  worst  score  for  station  I4.YA  could 
have  deteriorated  to  a considerable  extent, 

(3) Tho  effect  of  weather  variability  at 

various  stations  is  apparent  from  the  trends 
of  the  scores.  This  id  particularly  evident 
whon  the  scores  for  station  1^.YE  in  the 
subtropical  anticy^clone  are  compared  with 
those  of  station  ^YA  which  lios  in  the  North 
Atlantic  storm  track.  This  clearly  points 
to  the  necessity  for  separate  analysis  of 
th<'  for  Individual  stutlons  uux-ing 

tho  main  project, 

(I4.)  Tho  case  of  forecasting  varies  widely  for 


"Accuracy  of  Short-Range  Forecasting  Comparativo  Evalua- 
tions", GRD  Contract  Ho.  AF19(122)lii4.7,  0.  P*  Wadsworth, 
Project  Director,  Massachusetts  Institute  of 
Technology, 
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different  foreeast  items,  with  the  result 
that  the  more  difflcuHt  items  such  as  ww, 
h^hg,  and  tend  to  dominate  the  total 

scores.  This  fact  requires  further  study 
concerning  the  forecast  scoring  system, 
particularly  ^ith  respect  to  the  forecast 
limits.  It  is  also  Important  to  note  that 
tho  forecast  sydtem  and  tho  scoring  system 
applied  to  tho  main  project  will  be  some- 
what different  from  those  used  in  this  pilot 
tost.  For  example,  W,  ww,  RR,  and  h h 

c c 


will  be  forecast  in  greater  detail. 
Incidentally,  tho  forecasters  will  remain 
uninformed  of  tho  scoring  system  to  avoid 
forecast  bias. 


(5)  A more  rigorous  verification  does  not 
greatly  influence  tho  trend  of  tho  score, 
although  it  tends  to  incroaso  the  error. 

This  statement  can,  obviously,  bo  volid  only 
within  certain  limits  for  reasons  simllr'" 
to  thoso  mentioned  in  conclusion  2 above. 


The  pilot  tost  gave  tho  following  results  when  the 
upper  air  data  woro  analyzod; 

(1)  Tho  difference  between  12-  and  2l4.-hour 
forecasts  was  not  negligible  for  the  12-hour 
height  change  (Ah)  end  the  12-hour  tempera- 
turo  change  (i^T).  although  the  difference* 
between  12-  and  2l|.-hour  forecasts  in  wind 
speed  (PP)  and  direction  (DD)  was  still  slight. 
Figure  6 shows  the  effect  of  12-  and  2i|.-  hour 
forecasts  on  each  forecast  item  for  the 
various  schemes.  The  circled  points  indicate 
that  in  tho  particular  scheme,  the  item 
represented  by  tho  curvo  was  missing  from  the 
analysis  Information  which  the  forecaster 
had  available. 


(2)  Tho  average  error  noesro  f*or  Porecaater  I 
was  285?  in  the  12-hour  forecasts  and  345^ 

in  tho  2l+-hour  forecasts.  Forecaster  II 
averaged  in  12-hour  and  in  ?l4.-hour 
forecasts • 

(3)  The  effect  of  the  various  schemes  on 
each  particular  forecast  item  is  not  at 

all  clear.  However,  Figure  6 shows  that  for 
tho  2l, ‘hour  forecast  ofZi^H,  tho  nccurac*^ 
for  tho  schomos  in  which  the  hoight  was 
missing  was  considerably  loss  than  for  schemes 
in  which  the  height  was  glvon. 
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Fig.  6.  Effect  of  Various  Schemes  on  the 
$00-mb  Forecast  Items  for  12-hro 
and  24-hr.  Forecasts,  Circled 
Schemes  Indicate  that  the  Particu- 
lar Forecast  Item  Represented  is 
Missing  from  that  Scheme, 


13. 


(i^.)  The  over-all  deterioration  of  forecast 
accuracy  appears  significant  only  for  scheme 
E.  Tho  average  per  cent  error  for  schemes 
A and  B is  2$%  and  27?f,  for  schemes  C and 
D it  is  3lJ^  and  30^,  respectively,  and 
scheme  E the  averoge  por  cent  errors,  roacned 
a maximum  of  365^. 

Figure  7 comblnos  both  12-  and  2l|.-hour  fore- 
casts and  represents  the  por  cont  errors  for 
each  station  and  scheme.  No  definite  deterior- 
ation trend  is  evident.  Por  example,  the  por 
cent  errors  for  stotion  74.7  seem  to  decrease 
with  decreased  Information,  whereas  stations 
304.  and  4-^  show  the  opposite  trend. 

As  was  done  in  the  case  of  tho  surface  test, 
a relative  frequency  distribution  was  deter- 
mined for  four  groups  of  por  cont  errors  and 
for  all  schemes.  Tho  distribution,  is  shown 
in  Figure  8.  No  pronounced  trend  seems 
evident  in  this  distribution. 

Prom  the  above  results  the  following  conclusions  are 
stated  and  discussed: 

(1)  There  la  a pronounced  difference  between 
the  12-  and  24.-hour  forecasts  of  temperature 
change  and  height  change.  Whether  this  fact 
is  the  r,.;sult  of  the  verification  system 

or  of  tho  variability  of  tho  fcrocast  parameter 
in  question,  is  not  certain  at  this  time. 
However,  there  scorns  to  bo  no  reason  why 
tompornture  and  height  should  vary  to  an- 
groctor  extent  than  wind  direction  and  sp.^.^d. 

(2)  Tho  average  scores  of  tho  two  forecasters 
were  very  similar  even  when  divided  into 

12-  and  24--hour  forecast  categories.  This 
conclusion  sco.-ns  to  go  along  with  similar 
results  obtninud  in  tho  surface  pilot  test 
analysis i 


(3)  Score  aotorioration  as  a function  of  the 
scheme  used  does  not  aoom  to  hold  ns  well 
as  tho  case  in  the  surface  pilot  tost.  Only 
on  tho  average  did  scheme  E of  tho  upper  air 
tost  show  any  marked  deterioration.  Vi/hen 
the  data  wore  broken  up  into  individual 
stations,  tho  deterioration  as  n function  of 
tho  scheme  showed  opposite  effects  in  many 

COSOS. 


STATION 


0 - 19  20  - 39  l^-O  - 59  6o  - lOO 

FORECAST  ERROR  GROUPS 
(Errors  in  %) 


Fig.  8.  Frequency  Dlstributiona  of  uppoi*  k1i-  rorecasL 
Errors  (Errors  In  , Schemes  A Through  Eo 


In  addition  to  tho  score  analysis  of  the  surface 
and  upper  air  pilot  tests,  a time  study  was  also 
made.  Table  V givos  tho  average  time  in  minutes 
spent  in  spotting,  analyzing,  and  forecasting  for- 
each  scheme* 

Table  V.  Average  time  in  minutes  spent 
in  tho  spotting  of,  analyzing 
of,  and  forecasting  from  the 
map  series. 


Scheme 

m 

B 

D 

E 

Spotting 

Analysis 

forecast- 

1 

102 

9k 

96 

69 

6l 

6i 

k7 

T 79 

75 

5^ 

m»m 

An  inspection  of  tho  averages  listed  in  Table  V shows 
that  schemes  D and  E took  30^  loss  time  than  schemes, 
A,  B,  and  C.,  Obviously,  a major  faccor  in  this  30JC 
difference  is  the  decreased  time  spent  in  spotting  tho 
reduced  inf ormation.  Analysis  time  remained  fairly 
constant,  whereas  forocenting  time  was  greatest  for 
scheme  E and  least  for  scheme  D,  This  last  result  is 
obviously  duo  to  tho  fact  that  a for cod  response  is 
required  of  tho  forocastcr.  Ho  naturally  objects  to, 
and  consequently  hesitates  in  forecasting  from  a map 
that  he  feels  furnishes  him  with  insufficient  infor- 
mation to  perform  tho  task  at  hand. 


tentatively  appraising  tho  optimum  over-all 
conditions  for  spotting,  analyzing,  and  forecasting 
with  due  conaldcration  of  tho  forecast  scores,  it 


f vs  n ^ V>  O 4"  o r>  ^ ry, 

w v.^x«>4  V 


information  that  reduces  the  time  to  the  greatest  extent 
without  significantly  affecting  the  forecast  accuracy. 


As  a result  of  tho  pilot  tost  the  ft  llowlng  general 
remarks  can  bo  made: 


(1)  One  of  tho  primary  results  of  the 
pil  ot  tost  was  to  give  tho  project  members 
a ’'fool”  for  tho  procedure  and  tho 
difficulties  involvod.  For  instanco,  as 
a result  of  going  thi’ough  the  five  surface 
schemes,  it  was  decided  to  add  a sixt''. 

(F)  scheme,  since  it  was  thought  necessary 


ra.-.v  y , 


> * %/ 
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to  start  off  the  analysis  proceduro  by 
forcing  the  forecaster  to  drew  air-flow 
linos  without  tho  aid  of  prcssuro  data« 

(2)  Tho  tost  clearly  brought  ou<t 

that  station  location  and  soraotiiries  the 
length  of  tho  forocast  should  bo  consldored 
as  paramotors  when  tho  main  tost  is  analysed. 

(3)  Tho  most  perplexing  difficulty  is  tho 
choice  of  the  verification  limits.  This 
problom  is  now  boing  thoroughly  reviewed 
by  staff  mombors  not  directly  ccnnocto.^ 
with  the  project.  Fortunately,  the  work  in 
■enalyging  and  forecasting  will  not  bo 
dolayod  because  of  this  difficulty,  since 
th.:  f >rcc-;3tcr  will  not  be  informed  of  the 
verific'tion  limits.  In  addition,  in  case 
tv.o  vurificotion  systems  are  decided  upon, 
tho  punch-card  svstem  to  be  used  will  allow 

space  for  two  such  vorif icati'n  schemes. 

(li)  An  important  part  of  tho  tost  was  tho 
time  study.  In  deciding  the  over-all 
feasibility  of  one  information  scheme 
•against  another,  tho  amovmt  of  time  saved 
in  spotting,  analyzing,  and  forecasting 
should  represent  an  important  factor, 

4»2  Prognostic  Chart  Exporimont 

On  11  August  1952,  Mr,  David  Jones  initiated  an 
experiment  in  tho  regular  synoptic  meteorology 
laboratory  classes  in  which  37  undergraduate  and  special 
(Air  Force  officer-trainee)  students  participated. 

This  tost  consisted  of  (1)  dividing  the  group  into 
two  sections,  each  of  which  worked  independently 
without  opportunity  for  Intercommunication,  (2)  giving 
one  section  prepared  prognoetlc  charts  (on  the  forocast 
forms)  that  cansistod  cf  tho  actual  positions  of  fronts 
and  pressure  centers  as  detormiuod  by  pro-analysis 
(by  nrojoct  members)  of  tho  next  two  maps  ( i.c.-  .'.2- 
and  ^-hour  "prognostic”  positions),  (3)  requiring  the 
other  soctlon  to  construct  its  own  prognostic  charts, 
and  (I|.)  requiring  all  students  to  make  12-  and  2ij.-hour 
forecasts  for  tho  following  stations: 

(1)  Moosonoo,  Ontario,  836 

(2)  International  Palls,  Minnesota, 7i<-7 

(3)  Caribou,  Maine,  712 


(l^.)  Chicago  (Joliet),  Illinois,  536  (JOT) 

(5)  Buffalo,  New  York,  528 

(6)  Nantucket,  Hossachusette,  5o6 

(7)  Kansas  City,  Missouri,  (Port  Lonren- 
worth,  Kansos),  14t-6  (PLV) 

(8)  Nashville,  Tennessee,  32? 

(9)  Katteras,  North  Carolina,  30I4. 

(10)  Brownsville,  Texas,  250 

(n. ) Miami,  Florida,  203 

(12)  Kindi. ty  fiPB,  Bermuda,  016 

Forecast  itoms  ir'.cludod  in  this  tost  were:  sky 
cover,  wind  direction  and  speed,  present  weather, 
'•oiling,  visibility,  precipitation  amount,  temper  a tore, 
dew  point,  700-rab  temperature,  700-mb  height,  700-mb 
wind  speed  end  direction.  The  maps  used  consisted  of 
a six-map  series  of  North  American  soa-levei  and 
700-mb  charts  of  12-hour  continuity.  They  were  obtained 
from  Weather  Training  Supplies,  Inc.>,  Cambridge, 
Massachusetts,  end  covered  the  period  from  l835Z  on 
15  December  19^8  through  0635Z  on  I8  December  19l|-6. 

A second  test  was  conducted  under  the  same 
condition.s  but  using  Northern  Hemisphere  sea-level 
maps  of  24"hour  continuDty,  which  hu>’.  been  obtained 
from  the  Department  of  Meteorology,  Now  York  University 
for  the  period  from  1300Z  on  1 March  1937  throvigh 
13OOZ  on  6 March  1937.  Only  2I4. -hour  sea-level  forecasts 
for  the  following  stations  were  required: 

(1)  El  Paso,  Texas 

(2)  Cape  Race,  Nevfoundland 

(3)  London,  England 
(if)  Algiers,  Algeria 
(5>  Moscow,  U.  S.  S.  R. 

(6)  Bukhara,  W.  Uzbek  S,  S.  R. 

(7)  Irkutsk,  Irkutsk 


(8)  Canton,  Chino 

(9)  Dutch  Harbor,  Alaska 

(10)  Honolulu,  Hawaii 

Forecast  items  Ineludod  In  this  test  were  sky  coverage 
wind  direction  and  speod,  present  weother,  eelllng, 
visibility,  past  weather,  end  temperat\u*o# 

Tl\e  varirication  systems  used  In  both  testa  ore 
similar  to  that  usod  in  the  pilot  tost. 


4 *21  Results  of  Prognostic  Chart  Experiment 

The  37  students  were  divided  into  two  groups  of 
18  and  19 »respecbively,  aid  placed  In  different  rooms. 

The  groups  were  Instructed  to  avoid  discussion, 
comparison,  or  collusion  of  any  kind  while  analyzing 
charts  and  making  forecasts.  Group  A was  Issued  stirface 
prognostic  charts  for  12  and  2k  hours  from  the  time  of 
the  maps  the  students  -./Gre  to  analyze.  Group  B was 
issued  the  same  pre-plott<=  i charts  for  analysis  but  no 
prognostic  charts.  Half  way  through  the  experiment 
this  procedure  was  reversed;  Group  B was  issued  the 
prognostic  charts,  Group  A was  not.  That  step  was 
taken  to  cancel  whatever  forecasting  superiority  one 
group  might  have  over  the  other. 

A totnl  of  forecasts,  of  13  items  each,for  12 
stations  was  obtained  from  the  first  ^I.  American  and 
N.  Atlantic) 6-map  series;  2iC  forecasts,  of  8 items  each, 
for  10  stations  from  the  seoond(N.  Hemisphere ) map 
series.  For  the  f irst  series  the  )|V|  forecasts  were 
composed  of  four  groups  of  111  forecast  a;  12  -houi*  foro* 
castswlth  prognosis,  12-hcur  forecasts  without,  24~bour 
forecasts  with,  and  2ii-hcur  forecasts  without.  In  the 
second  series  the  218  forecasts, for  which  only  2i|.-hour 
continuity  was  available,  were  conposed  of  two  groups 
of  111  with  and  10?  without  prognostic  charts.  The 
forecasts  were  verified  on  a ’'percent-correct”  basis  and 
mean  group  scores  computed,  showing  the  following  results: 


First  6-Map  Series 

Mean  score  12-hr.  forecast  with  prog. 65o9?^ 

12-hr.  forecast  without  pros:. . . .6k»7jC 

Superiority  prog  over  no-prog  12-hr  forecast 1.2^ 

Mean  score  24-hr  forecast  with  prog 60,25f 

2l4.-hr  forecast  v/lthout  prog., «$7.75^ 
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Superiority  prog  over  no-prog  2i|.-hr  forecast 
Second  6~Map  Series 


2.55^  I 

I 

I 


Mean  score  24.-hr  forecast  with  prog.. 67*2^  i 

i 

24.-hr  forecast  without  prog .68«66C 

Superiority  no-prog  over  prog iM 


Statistical  Analysis  of  Results  of  Prognostic 
^periment.  Because  the  differences  in  scores 
between  the  prog  and  no-prog  charts  were  small, 
it  was  necessary  to  determine  whether  or  not  these 
differences  were  statistically  significant.  No 
significance  test  was  made  for  the  results  of  the 
second  map  series  for  reasons  which  are  outlined 
in  section  4-.213.  A frequency  distribution  for 
the  14^4  forecasts  of  the  first  map  series  was 
complied  and  a frequency  polygon  constructed. 

The  first  frequency  polygon  was  revised  with 
seemingly  unnecessarily  small  class  intervals  for 
reasons  discussed  in  section  4-»212.  The  frequency 
polygon  is  shown  in  Figure  9»  Standard  devia- 
tions computed  by  the  "short”  method  were  as 
follows : 


Stydard  Deviations  of  12 -hour 
Forecast  Moores 


with  prognostics  charts  ^ 

without  prognostic  charts  ,o_mO$% 


Standard  Deviations  of  2h.-hour 
Forecast  Scores 


with  prognostic  chrrts 
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Pig.  9*  Frequency  Polygon  for  Unadjusted  Forecast 
Scores.  Prognostic  Chart  Experiment. 
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where  ® end  er  ® are  the  wariancee  of 
a o 

forocastlng  sooros  In  Group  A and  Group  B, 
respectively,  and  and  isitjj  the  number  of 
participants  in  Group  A and  Group  B,  respectively. 

The  means  differed  by  1.63  standard  deviations 
of  the  differ  nces  of  the  means  for  the  12-hour 
forecasts  and  2.65  standard  deviations  of  the 
difforonces  of  the  means  for  the  2l4.-hour 
forecasts.  Taking , conservatively,  3.0  standard 
deviations  as  a basis  for  determining  significanee , 
the  percentage  differences  between  tho  forecasts 
with  prognostic  charts  and  those  without  were  not 
significant.  However,  the  undorlying  assumption 
that  the  a hove  forecasts  wore  uncorrelated  was  not 
justified  since  a low  soore  by  Group  A was 
accompanied  by  a low  score  by  Group  B in  the  case 
of  difficult  forecasts;  hlg^i  scores  by  both  Groups 
In  the  case  of  relatively  easy  forecasts.  That 
fact  led  tc  the  development  of  a more  complicated 
statistical  significance  tost. 


4.212  Correction  Factor  Applied  to  Statistical 
Results  of  Prognostic  Chart  Experiment.  Five 
factors  contributed  to  the  variation  of  the 
individual  f oracasters * ocerbs  In  tho  prognostic  ©hart 
experiment:  (1)  superior  forecasting  abillt:  f one 

Group;  (2)  presence  or  abwenco  of  prognostic  charts; 
(3)  difficulty  of  the  synoptic  situation;  (4)  indivl 
dual  forecasting  skill;  and  (5)  random  influences. 
Variotions  in  scores  because  of  (1)  were  eliminated 
by  computing  mean  scores  for  each  foi-eoast  type 
and  by  rotating  issuance  of  prognostic  charts  between, 
the  two  Groups.  This  loft  (2),  (3),  (i4)» 

(5)  responsible  for  producing  the  differences  in 
the  moan  scores  made  under  each  forecast  type. 


The  effect  of  the  difficulty  of  different  fore- 
casts is  shown  by  the  following  example:  The  mean 

score  for  all  participants  for  the  ^-hour  fore- 
cast from  the  third  map  was  50.0^,  whereas  the 
mean  score  for  all  participants  for  the  24 -hour 
forecast  from  the  sixth  map  was  68.65^--  a 
difference  of  16.6^0.  II  was  observed  that  if  all 
six  forecasts  of  each  type  could  be  reduced  to  t he 
same  degree  of  difficulty,  tho  remaining  difi'erence 
could  bo  attributed  only  to  (2),  (4)>  and  (5). 

Tho  difficulty  of  a forecast  can  be  measured  by 
the  difforanco  betwoenthe  mean  score  for  all 
forecasts  and  tho  mean  scoi'e  for  a particular 
forecast.  If  this  differonco  is  added  to  each 
indivlduol  forecaster’s  score  to  yield  a corrected 
score,  this  corrected  score  should  be  independent 
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r^f  «- 


cf  the  difficulty  of  the  forecast.  Fui?themjore, 

It  seems  roasonable  to  assume  that  no  correlation 
exists  between  the  corrected  scores  oT  Group  A 
and  those  of  Group  B.  With  this  assumption, 
corrected  scores  were  computed  and  their  standard 
deviations  found  from  the  standard  deviatlor  ' the 
uncorroctod  scorec  by  a formula  derived  fay  Dr.  Hans 
Panofsky,  The  derivation  of  the  formula  is 
reproduced  and  explained  in  t he  Appendix  to  this 
Progress  Report.  The  standard  deviations  of  the 
corn *c ted  scores  wore  as  follows; 

Standard  Deviations  of  Corrected  12-Hour 
Forocast  Scores 


with  prognostic  charts 4»1^ 

without  prognostic  charts 


Standard  Deviations  of  Corrected  2lj-Hour 
Forocost  Scores 


tith  prognostic  charts 

without  prognostic  charts 


The  fact  that  these  vnluos  are  considerafaly 
smaller  than  the  standard  deviations  of  uncorrocted 
foocenst  scores  Indicates  a great  influence  of 
the  relative  forecast  difficulty  on  bho  variation 
of  the  forecast  scores.  Quanti^'atively,  the 
varlanco  because  of  the  variation  of  the  dlfferencof 
of  forecast  scores  accounts  for  approximately  half 
the  total  variance  of  the  forecast  scores. 

Prom t ho  standard  deviations  of  the  corrected 
forecast  scores,  c'  was  rocoriputed.  The 
differences  in  the  m&ons  of  scores  of  Groups  A 
and  B woro  2,l6  cr-^  for  the  12-hour  forecasts  and 
lu.Ol  C7-.  for  the  2£t-hcvi'  fcrscnntq.  According 
to  thoso'^f iguros , the  score  difference  in  favor  of 
the  prognostic  chart's  effect  on  the  forecasts 
Is  significant  in  the  case  of  the  2l4.-hour  forecasts, 
and  possibly  so  in  the  case  of  the  12-hour  fore- 
casts, depending,  of  course,  on  whether  the  2- 
or  3~atandQrd  dovlatlon  limit  is  used  as  a 
criterion  of  significance.  The  fact  that  the 
differ  nee  in  scoros  is  statistically  significant 
may  have  little  practical  value  since  the  difference 
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in  th&  case  of  12-hour  forecasts  w as  only  l.l^^ 
and  in  the  case  of  2l^.-hour  forecasts  2.5j«* 

The  frequency  of  corrected  scores  In  any  one 
class  interval  of  the  frequency  distribution  was 
greatly  increased,  so  that  class  Intervals  of 
3^1 03  in  3?lguro  9#  seemed  justified.  The  frequency 
polygon  constructed  for  the  corrected  scores 
is  shown  In  Pigure  10. 

li..213  Additional  Factors  Contributing  to  Hesults 
of  Prognostic  Chart  Experiment.  The  statistical 
results  shov.n  in  the  preceding  sections  are 
necessarily  based  on  the  assumption  that  the 
forecasts  by  each  man  were  made  independently. 
Although  there  was  a supervisor  in  each  laboratory 
at  all  times  throughout  tho  oxporlment,  assurance 
that  thoro  was  no  collusion  be twoon forecastors 
was  not  posltlvo;  in  fact  during  the  second 
(Northern  Hemlsphcro)  sarios  it  became  clear  that 
tho  requirements  outlined  in  section  i|.e21  wore 
not  being  followed.  Several  forecasters  admitted 
that  they  did  not  use  the  prognostic  chart  when 
making  their  forecasts.  Othore found  it  expedient 
to  retain  by  memory,  or  copv;  tho  prognostic  chart 
from  day  to  day,  Boeause  of  thoae  clrcumstancos 
it  was  felt  that  the  results  of  tho  second  series 
were  moanlngloss;  honco  no  statistical  significance 
testa  were  attempted.  Tho  laboratory  supervisors 
felt  that  such  was  not  tho  cose  during  tho  first 
(North  American)  series,  however;  honce  the  results 
may  be  Indicative  of  tho  effect  of  a ”100jC  accurate’ 
prognostic  chart. 

Statistical  significance  in  goneral  moans  that  a 
conclusion  is  likoly  to  bo  valid  for  a population 
out  of  which  tho  givon  sample  has  boon  drawn  -t 
random.  In  particular,  this  moans  that  tho 
experiment  doscrlbod  has  only  shown  that  prognostic 
charts  are  useful  for  tho  typo  of  porsorinol  tested, 
but  are  not  nocossarily  useful,  or  may  be  moro 
useful,  for  oxporioncod  forocastors  in  tho  field. 

Because  of  the  dlfricultles  involved  in  supervising 
a largo  laboratory  group,  tho  experiment  is  bolng 
repeated  at  prosont  using  a small  nvunbor  of  mon 
doing  u largo  munbor  of  forocasts.  Furthormoro , 
it  is  plannod  to  verify  only  those  forecast 
items  that  would  doflnitoly  bo  affected  by  the 
presence  or  absence  of  a correct  prognostic  choi’t. 
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Pig.  10.  Frequency  Polygon  for  Adjusted  Forecast 
Scores.  Prognostic  Chart  Experiment. 
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(1)  The  project  team  Is  anticipating  a ^rislt9 
early  in  October  by  LCDR  Paul  M.  Wolff  and  LTvO  Robert 
R,  Dickson,  of  Bureau  of  Aeronautics  Project  /iROWJs  . 
to  State  College.  The  purpose  of  this  visit  will  oe 
to  eliminate  evidences  of  cross-purposes  in  the  / 

investlgotion  end  to  acquaint  LTJO  Dickson  with  the 
details  of  previous  and  planned  work  in  anticipation 
of  his  collaboration  in  tho  analysis  ctnd  forecasting 
of  the  final  test  data.  LTJG  Dickson's  participation 
will  bo  treated  on  a correspondonco  basis,  in  the  some 
manner  as  that  of  Mr.  Simmons. 

6,0  Fiscal  Information 

July  1,  1952  through  Soptombor  3^,1952 


Salories.  ..........  .. 

Wagos 1,132.50 

Overhead  on  V#agos  and  Salaries.  1,I|20.62 

Supplies  and  Materiols  . 116.22 

Equipment 1^95*19 

Trovol.  114-9.18 

7.0  Authorship 


The  writing  of  this  report  was  coordinated  h" 
the  Assistant  Project  Supervisor,  Mr.  Moyer,  with  CxiO 
active  assistance  of  Mr.  Van  dor  Hcven,  who  wrote  up 
tho  results  cf  the  pilot  test,  and  '.h».  Jones,  who 
described  the  results  of  tho  prognostic  chart  experiment. 
Tho  drawings  wore  prepared  by  the  two  last-montlonod 
mon;  and  Dr.  Panofsky  collaborated  with  Mr.  Jones  in 
preparing  tho  Appendix.  Tho  editorship  of  the  paper 
was  shared  by  Dr.  Noub^rger  and  Mr.  Moyer,  and  Mrs. 

Mary  Wagner  did  all  tho  secretarial  work  as  well  as 
the  proof-reading. 
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APPENDIX 


Derivation  of  the  Equation  Prom  Which  the  Variance 
of  the  Corrected  Forecast  Scores  was  Computed. 


The  variance  from  an  arithmetic  mean  la  defined  by 
the  relation 

^ 2(X  - 7.)^ 

N 

where  X la  each  participant's  score  for  each  for',  .at, 
X Is  the  mean  score  for  a particular  forecast  type 
(12-  or  2l;.-hour  forecast,  with  or  without  prognostic 
chart),  and  N is  the  total  number  of  forecasts  in 
each  forecast  type  (111), 


X must  be  adjusted  by  adding  a correction  factor, 
necessary  to  reduce  all  forecasts  to  the  same  degree 
of  difficulty.  This  correction  may  be  designated 
by  C and  la  defined  as  ^ - Xj,  where  X Is  the  mean 
of  all  forecast  scores  for  each  of  the  12-  and  24-hour 
groups  of  six  forecasts,  and  Xj  is  the  average  for 
any  one  of  the  six  forecast  scores  (scores  having 
the  benefit  of  prognostic  charts  averaged  together  with 
those  not  having  the  prognostic  charts),  one  for 
12-hour  scores,  the  other  for  2l4--hour  scores.  Then 
for  each  forecast  type,  the  corrected  variance,  which 
may  be  designated  by  becomes 

2 2[(X  + 5)  - X]2 

O’  (1) 

or 

— ’*"'2 
p A I.  V A — a;  o j 

(2) 

N 

Expanding  the  right  terra  of  equation  (2), 

2 2(X  - X)2  Z(x  - X)?  2C2 

CTc^  = + 2- 4 (3) 

N N N 


The  first  torn  of  the  right  side  of  equation  (3)  is 
the  variance  of  the  uncorracted  scores  which  may  be 
designated  by  , The  second  and  third  terms 

represent  the  correction  to  bo  added  to  to 

arrive  at  the  variance  of  the  corrected  forecast  scores. 
The  summation  sign  in  the  second  and  third  terms  essentia.1- 
ly  reprosonts  two  succossivo  summations:  (1)  , the 

sum  of  men  within  each  laboratory  making  ono  forecast; 
and  (2)  Zj  , tho  siaa  of  all  six  forecasts  cither  with 
or  without  prognostic  charts,  in  this  oxporlmont 
is  over  the  six  map-days;  2-  is  over  the  l8  or  19 
mon  in  oach  laboratory. 


Rewriting  aquation  (3) 


+ 2— -A 


N 


ih) 


The  forocast  difficulty  correction  factor, 
constant  rogardloss  of  the  number  of  men  In 
laboratory.  Thcroforo,  it  is  constant  .vith 
to  2j  and  equation  (1+)  bacomes 

, . Z,S,tSj(X  - X)) 

0.2  . + 2-——^ * 


^i,  is 

oach 

rospoct 


(5) 


The  portion  of  the  second  term  within  the  brackets 
in  (5)  may  bo  expanded:  2-;X  - 2jX.  The  term 

is  the  summatioh  of  the  average  scores 
for  each  of  the  four  forccost  typos  for  the  six  map 
days  ond  may  be  dosignntod  by  mX,  where  m is  either 
18  or  19  in  this  e xperiment,  Tho  term  Xj,  , 
tho  average  score  jT  oach  forocast  typo  for  any 
particular  forecast,  is  defined  as 


X.  = 


2 47. 


m 


(6) 


Rfl 'Vr*T' fl  n rr  in  rj 
- — 9 


The  terms  2^Xi  (within  brackets  in 

equation  5)  may  bo  Expressed  as  follows: 


(7) 


2j(X  > X)  mXj^  - rciic  = m(Xj^  - X)  (S) 

Tho  diffarcnco  between  the  avorago  in  oach  laboratory 
score  for  ono  forocast  and  tho  average  for  tho  six 


forecasts  may  bo  oxprosaod  by  . Xi.  Tho  differonce 
is  coiistant  for  oach  forocnst  and  mQy*be  doslgnatod 
as  , Thorofore, 

2j(X  - X)  = m(Xi  - X)  = - mCi  (9) 

substituting  (9)  into  (5) 


S^C^mCj 

K 


With  this  equation  tho  variance 
scores  was  computed. 


N " 

of  tho  corrected 
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